Assessing Schematic Knowledge of Introductory Probability Theory

The ability to identify schematic knowledge is an important goal for both assessment and instruction. In the current paper, schematic knowledge of statistical probability theory is explored from the declarative-procedural framework using multiple methods of assessment. A sample of 90 undergraduate introductory statistics students was required to classify 10 pairs of probability problems as similar or different; to identify whether 15 problems contained sufficient, irrelevant, or missing information (text-edit); and to solve 10 additional problems. The complexity of the schema on which the problems were based was also manipulated. Detailed analyses compared text-editing and solution accuracy as a function of text-editing category and schema complexity. Results showed that text-editing tends to be easier than solution and differentially sensitive to schema complexity. While text-editing and classification were correlated with solution, only text-editing problems with missing information uniquely predicted success. In light of previous research these results suggest that text-editing is suitable for supplementing the assessment of schematic knowledge in development. Schematic Knowledge of Probability 3 Assessing Schematic Knowledge of Introductory Probability Theory Knowledge is many things to many people. de Jong and Ferguson-Hessler (1996) have commented that knowledge has been conceptualised as domain-general and domainspecific, concrete and abstract, implicit and explicit, formal and informal, elaborated and compiled, declarative and proceduralized, conceptual and procedural, unstructured and structured, tacit or inert, strategic, situated, schematic, as knowledge-acquisition knowledge, as metaknowledge – and the list goes on. It would seem that knowledge, like intelligence, is a rather nebulous construct. However, within this quagmire a small number of approaches to conceptualising knowledge have received almost consensual support. The declarativeprocedural framework is one such approach for which evidence has amassed across numerous domains (Anderson & Schunn, 2000; Benaroch, 2001; Cohen, Poldrack, & Eichenbaum, 1997; Kirasic, Allen, Dobson, & Binder, 1996; Low & Over, 1992; RittleJohnson, Siegler, & Alibali, 2001). How we conceptualize knowledge not only determines the nature of our instructional strategies but also how we assess it. In the current study we adopt the general declarative-procedural framework to explore the assessment of schematic knowledge of probability theory. This element of statistics is particularly difficult for undergraduate students to master (Konold, 1995) and there are ongoing calls for more innovative ways to assess statistical knowledge (Garfield, 1994). Schematic knowledge is highly context specific (Quilici & Mayer, 2002). It contains declarative knowledge (also referred to as conceptual knowledge, de Jong & Ferguson-Hessler, 1996; Rittle-Johnson et al., 2001) and procedural knowledge, in addition to situational knowledge that provide cues to when knowledge should be used (Cheng & Holyoak, 1985; de Jong & Ferguson-Hessler, 1996). A traditional method used to assess schematic knowledge is to have students classify pairs of problems in terms of structural relatedness. We compare this to a relatively new Schematic Knowledge of Probability 4 method that has received little attention in the literature called text-editing. Before we describe the text-editing task, a brief overview of the declarative-procedural-schematic framework is presented. Declarative knowledge. Declarative knowledge is commonly considered as knowing-what, and procedural knowledge as knowing-how. The declarative-procedural framework has been at the heart of numerous information-processing theories of knowledge-acquisition (e.g., Anderson, 1982; Anderson & Milson, 1989; Anderson & Neves, 1981; Anderson & Schunn, 2000). During the early stages of skill acquisition, declarative information can be used with general problemsolving procedures in an interpretative way (Anderson, 1990). For example, a student learning to solve area-of-rectangle problems in a mathematics course may initially encode declarative information relating the length of adjacent sides of a rectangle to the surface area. The student is likely to have had some experience solving other mathematics problems and will also have access to general problem-solving strategies used in everyday reasoning. These additional experiences can be brought to bear on the newly acquired declarative information about rectangles to facilitate success without the student necessarily possessing specific solution strategies or procedures (Anderson, 1982; Anderson & Neves, 1981; Rumelhart & Norman, 1981). Two examples of general problem-solving approaches that have received considerable attention in the literature are working forward and working backward strategies. An example of a working forward reasoning strategy would be making a supposition that a particular formula for solution is appropriate from the start, and then following the ensuing inferences through to their logical end (Byrne & Handley, 1997; Rips, 1989). An example of a working backward strategy is the means-ends approach, which is essentially a goal directed, searchSchematic Knowledge of Probability 5 oriented process frequently used by novices (e.g., Gick, 1986). The idea behind this procedure is that differences between the goal-state specified by the problem and successive problem states obtained during the course of solution are analysed in order to choose the next best move or sub-goal (Sweller & Levine, 1982). While means-ends analysis is an efficient and flexible problem-solving strategy in novel situations where specific procedures or schemas are lacking, it is resource-demanding because a comprehensive search of memory is necessary at each sub-goal (Gick, 1986; Sweller, 1988, 1989). In fact, general problemsolving procedures can make such high demands on working memory resources that they can actually interfere with learning (Van Gog, Paas, & Van Merriënboer, 2004). For instance, Sweller and Levine (1982) showed that maze learning was inhibited by means-ends approaches. Similar inhibitory effects on learning have been shown in algebraic word problems (e.g., Cooper & Sweller, 1987; Ward & Sweller, 1990; Zhu & Simon, 1987). The inference from these studies is that general problem-solving consumes working memory resources that would be better spent on learning the structure of the task at hand (Sweller, 1988, 1989, 1993; Sweller, Chandler, Tierney, & Cooper, 1990; Sweller, Mawer, & Howe, 1982; Woltz, 1988). Procedural knowledge and learning. A common theme in many cognitive learning theories is that people construct new knowledge by “assimilating” their current experience with what they already know or believe to be true about the world (Chen, 1999; Cook, 2001; Gick & Holyoak, 1983; Piaget, 1950). As experience in a domain increases, related declarative facts and specific solution strategies are linked – they become proceduralized. Instruction is often premised on this understanding. That is, while proceduralization means that the use of knowledge becomes less dependent on domain-general reasoning, proceduralization has also been shown to facilitate further Schematic Knowledge of Probability 6 learning (e.g., Rittle-Johnson et al., 2001). Two aspects related to this are problem structure and context. We consider each briefly. Structure: One of the main objectives of instruction is to help students to focus on the structure of a domain and to draw their attention away from the more transient surface details of a particular problem. For instance, altering the format of instructions of learning tasks to be less goal oriented has been shown to facilitate learning because it encourages learners to be more sensitive to the structure of the task (Chandler & Sweller, 1991; Sweller, 1993; Sweller et al., 1990; Sweller & Levine, 1982; Van Gog et al., 2004) and less focused on surface details (Quilici & Mayer, 1996, 2002). Imagery has also been used to improve understanding of mathematical structure. An important feature of visual-spatial presentation on knowledge acquisition has to do with how elaborated learning can facilitate the recall of important structural features of problems. However not all types of visual-spatial representations have been shown to be equally effective in improving mathematical achievement (Novick, Hurley, & Francis, 1999). Hegarty and Kozhevnikov (1999) suggested that the use of schematic imagery (i.e., pure relationships depicted in a visual-spatial scheme devoid of concrete images) was positively correlated with mathematical performance, but that there was a negative correlation with use of pictorial imagery (i.e., concrete images in the mind). Pictorial imagery was associated with poorer performance because it takes the problem solver’s attention away from the problem structure. Context: Presenting problems in a familiar context has been shown to enhance knowledge acquisition and performance while at the same time reducing working memory demand (Quilici & Mayer, 1996, 2002). For example, Carraher, Carraher, and Schliemann (1985) studied Brazilian children who, for economic reasons, often worked as street vendors (see also, Nuñes, 1994). Mathematical problems were either embedded in real-life situations, Schematic Knowledge of Probability 7 where appropriate procedures and strategies for using declarative knowledge were in place (i.e., vending), or in an academic context, where, for this population, procedures were less developed. The children’s performance was significantly better in the former problems. Schematic knowledge. A more extended conceptualisation of the organisation of knowledge can be found in the concept of a schema. Schemas have been defined in many ways. Within mathematics, Cooper and Sweller (1987, p. 348) defined a schema as "...a construct that allows problem solvers to group problems into categories in which the problems in each category require similar solutions". Similarly, Sweller (1989, p. 458) defined a schema as a "...cognitive construct that permits problem solvers to recognise problems as belonging to a particular category requiring particular moves for solution". Furthermore, schemas are useful because they serve to categorise knowledge according to the way in which it will be used (Sweller, Van Merriënboer, & Paas, 1998). Although originating from slightly different lines of research (Chi, Fletovich, & Glaser, 1981; Sweller et al., 1998), schema theory fits closely with Anderson’s (1982; 1990) declarative-procedural framework. A common finding in the expert-novice literature has been that experts’ organization of knowledge is qualitatively different from that of novices and that this difference is a function of experts’ greater access to elaborated schemas (Chi et al., 1981; de Jong & Ferguson-Hessler, 1986) that develop with experience in the problem domain (Sweller et al., 1998). Contextualized schematic knowledge reflects a deep structural understanding of the domain, but this high contextualization can also come at a cost. It may place constraints on the availability of certain information and in some cases can produce near-transfer failure – the unexpected failure to transfer knowledge from one context to a similar one in the same domain (Woltz, Gardner, & Gyll, 2000). The fact that people can solve mathematics Schematic Knowledge of Probability 8 problems presented in a familiar practical context but fail dismally when only the context is changed (Carraher et al., 1985), is a clear example of one of the disadvantages of contextualized knowledge. However, because schemas are highly contextualized, under most circumstances they can be accessed directly without the separate facts and procedures having to be retrieved, interpreted, and assessed for appropriateness piece by piece. The end result is generally a faster more efficient processing system (Anderson, 1982, 1990; Anderson & Neves, 1981). There is evidence to suggest that the development of knowledge and expertise in a domain does not necessarily move from declarative, to procedural, to schematic in a smooth monotonic fashion (e.g., Rittle-Johnson et al., 2001) and that declarative and procedural knowledge exists side-by-side with schematic knowledge (de Jong & Ferguson-Hessler, 1986). For instance, Savelsbergh, de Jong, and Ferguson-Hessler (2002) suggest that even though concepts in many domains (especially mathematically based domains) can be ordered hierarchically from concrete to abstract, it is often unclear what is abstract and what is concrete, and how such a distinction might change from one individual to another. Nevertheless, there is evidence to suggest that depth of knowledge does improve as knowledge becomes proceduralized, and that expertise increases with the acquisition and development of schemas (Chi et al., 1981; Chi, Glaser, & Rees, 1982). Mayer (1987) argued that developing an understanding of problem structure should be the primary instructional goal in teaching mathematics. Schematized knowledge reduces cognitive load, freeing resources for the acquisition of problem structure, which in turn has been repeatedly shown to facilitate near and far transfer of skills (e.g., Sweller et al., 1998; Van Gog et al., 2004). The ability to identify schematic knowledge is therefore a worthwhile goal for both assessment and instruction (Ngu, Low, & Sweller, 2002; Quilici & Mayer, 2002). Schematic Knowledge of Probability 9 Assessing Schematic Knowledge Various techniques have been used to assess schematic knowledge. Students have been asked to recall details of problems they have either seen or solved previously (Low & Over, 1989, 1990). Recall of only surface details of problems has been shown to indicate poor schematic knowledge, whereas recall for structural details indicates that schemas have been used (Low & Over, 1990; Ngu et al., 2002). Another method of assessing schematic knowledge is to have students categorise problems in terms of their perceived similarities. As with recall, it is assumed that persons with poor schematic knowledge of the domain will sort by surface similarities, whereas persons with better knowledge will consider deeper structural components (Littlefield & Rieser, 1993; Low & Over, 1989, 1990, 1992; Schoenfeld & Herrmann, 1982). The criterion often used to validate these techniques is the student's ability to actually solve problems. The approach to assessing schematic knowledge that we used in this study is called text-editing (Low & Over, 1989, 1990, 1992, 1993; Low, Over, Doolan, & Michell, 1994; Ngu et al., 2002). The idea is that the ability to identify what information from the text of a problem should be used, in what sequence, and through what operations, reflects an understanding of the problem structure and is therefore evidence of appropriate schematic knowledge. Text-editing requires students to isolate the necessary and sufficient components needed to solve word problems. The problems are designed so that they fit into one of three mutually exclusive categories: (1) there is sufficient information given to solve the problem; (2) there is sufficient information plus irrelevant information; and (3) information required for problem solution is missing. In the last two cases, students are also required to identify the irrelevant or missing information. To illustrate the nature of text-editing, consider the following problem which was used by Low et al. (1994) to train students in text-editing areaSchematic Knowledge of Probability 10 of-rectangle problems. The length of a rectangular window pane is twice its width. The area of the pane is 98 cm2. What are the dimensions of the pane? If we are to consider the schema required to solve this problem, knowledge that the area is equal to the length multiplied by the width would be one necessary component (Low & Over, 1989). If the width = a and the length = 2a, as given in the problem, then retrieval of an appropriate area-of-rectangle schema that integrates potential solution procedures and necessary declarative facts would facilitate solution. It is not a requirement of text-editing to actually solve the problem, however in this case there is sufficient information provided to do so. If the student does not possess an appropriate schema to solve the problem, the probability of making an error in the text-editing task is increased. Low and Over (1989) were able to reinforce their claims for the value of text-editing as a direct measure of schematic knowledge by demonstrating that performance on textediting was strongly correlated with problem solution. They demonstrated this in two ways: Firstly by showing that text-editing predicted as much as 90 percent of the variance in subsequent problem solution; and secondly by showing that failure in the text-editing section of their experimental procedure was almost invariably followed by failure in the problem solution section. While text-editing seems to be a necessary condition for problem solution, it is not sufficient. Various studies have shown that a significant proportion of students were able to correctly text-edit problems but then failed solution (Low & Over, 1989, 1990, 1992) – we explored this finding further in the current study. Low and Over (1990) went on to show that text-editing is correlated with recall memory for algebraic word problems and discrimination of whether problems are similar or different. In this second study, text-editing scores were also correlated with tests of general mathematical ability, even after allowance Schematic Knowledge of Probability 11 was made for contributions from verbal ability. The authors concluded that because the task requires an understanding of problem structure, text-editing can be taken as a measure of schematic knowledge. Previous research by Low and colleagues indicates that text-editing is easiest for problems where there is sufficient information and most difficult where the problems contain irrelevant information. The extent that this pattern of results generalizes to probability problems is an empirical question. The successful application of text-editing techniques in mathematics has paved the way for its application in other fields. Ngu et al. (2002) examined its usefulness as a training aid in the field of chemistry and concluded that its effectiveness depended on the learning materials. Our particular interest in this study was in the field of statistics. There are some obvious parallels between the fields of mathematics and statistics. The format in which statistics problems are encountered by students is typically similar to the format of algebraic word problems used in text-editing research (Low & Over, 1989, 1990; Mayer, 1982). Furthermore, statistical probability problems are based on a number of related but distinct classes that require different formulas and procedures for solution. As detailed in the Method section, the text-editing problems used in the current study are drawn from five problem classes (see Table 1). These classes can be differentiated on the sophistication of the schema involved. For instance, solving probability problems using the addition rule when events are not mutually exclusive (Class A-NM in Table 1) is a more complex instantiation of the basic addition rule with mutually exclusive events (Class A-M). Similarly, Class M-NI involved a more complex instantiation of the multiplication rule than Class M-I. These classifications also tend to coincide with typical teaching approaches in these areas – simpler, more specific problem classes are often taught first followed by instruction in the more complex, general form of the equation. The acquisition of appropriate schemas would therefore seem to be Schematic Knowledge of Probability 12 particularly advantageous to learning in this domain because it would enable appropriate differentiation and application of formulae, a task that new students frequently find difficult. Consequently, we expect text-editing performance to reflect the different levels of schema complexity and establishing that these expected effects are present is an important preliminary step to the investigation of the main aims of the study. Table 1 Rules and formulas used for text-editing and solution problems Problem No. Class Rule Formula Sectio n M S I TE 13 10 7 A-M Addition Rule Mutually exclusive events. P(A ∪ B) = P(A) + P(B) Sol NA 5 6 TE 8 12 2 A-NM Addition Rule Not mutually exclusive events P(A ∪ B) = P(A) + P(B) P(A∩ B) Sol NA 4 3 TE 14 3 5 M-I Multiplication Rule Independent events P(A ∩ B) = P(A) × P(B) Sol NA 10 9 TE 4 11 15 M-NI Multiplication Rule Not independent events. P(A ∩ B) = P(A) × P(B|A) Sol NA 8 7 TE 6 1 9 Comb. Combined rules: A-M or A-NM with M-I or M-NI Sol NA 1 2 Notes: Solution problems (Sol) were constructed to be isomorphic to text-editing (TE) problems; M = missing; S = sufficient; I = irrelevant; NA = problems containing missing information were not presented for solution. The main aim was to extend the application of text-editing to the assessment of schematic knowledge of probability theory, an area of statistics that has traditionally proved a considerable hurdle for students from a variety of disciplines (Konold, 1995). A subsidiary aim was to test whether the high R2 values (.80 .90) reported in earlier studies by Low and Schematic Knowledge of Probability 13 Over (1989, 1990) are robust to a change in methodology. Low and Over used the same irrelevant and sufficient problems in both the text-editing and solution sections of the studies. That is, participants were first asked to text-edit a set of problems containing irrelevant, sufficient, and missing information and then asked to solve a subset of those same problems (i.e., the ones containing sufficient and irrelevant information). Repeated presentation of the same problems could inflate the correlation between text-editing and solution. The second aim of the present study, therefore, was to determine the relationship between text-editing scores and solution scores when different (but isomorphic) problems were used in both sections. Observation of high R2 values in this more stringent situation would certainly encourage serious consideration of text-editing as a way of assessing schematic knowledge in statistics. The third aim concerned the comparison between the text-editing approach to the assessment of schematic knowledge and the more traditional problem classification approach wherein participants are asked to classify problems in terms of problem-relatedness (Schoenfeld & Herrmann, 1982). In their first experiment, Low and Over (1990) did not find any relationship between problem solution and performance on a problem-classification task, nor was there any relationship between text-editing and problem-classification. In their second experiment, where it was more clear that the problems to be classified were “similar or different in terms of underlying structure and not surface detail” (Low & Over, 1990, p. 68), a relationship between text-editing and problem-classification was found. However, the relationship between solution and classification was not investigated. Should text-editing provide incremental prediction of solution, it would constitute further evidence that textediting is a better technique for assessing schematic knowledge than some of these traditional techniques. Schematic Knowledge of Probability 14 Finally, the population we have chosen to draw our sample from is very diverse. It consists of university students pursuing majors ranging from science and engineering, to business and arts and hence there may be strong individual differences in exposure to training in mathematics and statistics. Although sample diversity is desirable in demonstrating generalizability of the text-editing method, these varying rates of exposure may actually serve to qualify our comparison with the studies of Low and her colleagues who have used samples less heterogeneous in experience. Therefore, tests of the main aims will also be conducted on a homogenous sub-sample of the more quantitatively inclined science and engineering students. Method Participants The participants were first and second year students enrolled in a compulsory introductory statistics unit at the University of Southern Queensland, Australia. Testing time was arranged so that students had just completed a training module on probability and were preparing for an assignment on the same topic. Of the 100 students who volunteered to participate, 10 either failed to complete any of the last section of the test or did not finish in the designated time, and were therefore omitted from the analysis. The remaining 65 males and 25 females completed the study in the one hour allocated. Students ranged in age from 18 years to 44 years with a mean of 22.3 years (SD = 6.50). Approximately 31% of the students came from the Faculty of Engineering, 29% from Commerce, 24% from Science, and 16% from Arts. Instruments and Procedure Each student completed a booklet containing three sections. In addition, students were asked to provide some biographical information regarding their student status and to note Schematic Knowledge of Probability 15 some of their self-perceptions about statistical probability. The sections of the questionnaire were as follows: Schematic Knowledge of Probability 16 Section I Text-editing. Section I consisted of 15 problems presented in the text-editing format used by Low and Over (1990). Students had to classify whether problems contained (a) sufficient information for solution, (b) irrelevant information that was not necessary for solution, or (c) missing information required to solve the problem. These problems (five for each text-editing category) were drawn from the curriculum of the introductory statistics unit in which the students were enrolled (University of Southern Queensland, 1993). All problems were vetted by teaching staff to ensure the problems covered information students were required to know for assessment. An example of a problem used from each of these categories follows: Missing Information. You have just bought 5 tickets in a local raffle. Your brother has bought 12 tickets. What is the probability that you or your brother will win? Sufficient Information. From an ordinary deck of 52 playing cards, one card is selected at random. What is the probability that it is a picture card (Jack, Queen, King) or a 2? Irrelevant Information. Two cards are drawn randomly from an ordinary pack of 52 playing cards. This is done with replacement. If 25% of the cards are diamonds, what is the probability of obtaining a 9 and a Queen in that order? To ensure wide coverage of the domain, problems were selected from five classes based on either simple or complex conjunction (A ∪ B) and simple or complex disjunction (A ∩ B). These classes, shown in Table 1, can be considered to be determined by the formula that would be implemented by an appropriately cued schema. Care was taken to ensure that each class consisted of a sufficient, irrelevant and missing problem although respondents were not made aware of this. The problems were presented in the same randomised sequence to all students. Again, we expect that problem classes are not of equal difficulty and we will Schematic Knowledge of Probability 17 take this into consideration in the analyses. Section II Classification of problem relatedness. In this section, participants classified problems in terms of similarity (i.e., same or different). Twenty problems selected from the sufficient and irrelevant problems used in Section I and Section III (described below), were presented in ten pairs. Five of the ten pairs contained problems with different structural details (or formulae required to solve them), and five pairs contained similar structural details. Each member of a pair differed in terms of surface details so as to reduce the possibility of students making a comparison solely on this basis (Low & Over, 1990). Students were told at the beginning of the session that classification of relatedness should be made in terms of the strategies and procedures necessary to solve the two problems. Section III Solution. Section III required participants to solve ten problems based on those from Section I that contained sufficient and irrelevant information. Problems missing information were not used in this section because they do not contain sufficient information for solution. These problems were altered so that while they used the same structural formats as those from Section I, the surface details were different (i.e., they were developed to be isomorphic). Table 1 indicates the text-editing problem from which each solution problem was derived. Students were asked to show their working. General Procedure Testing was conducted in group sessions. The instructions for each section were read to the students at the start of the session so that on completion of one section students did not need to wait for others to finish. The limited testing time necessitated this procedure. Written instructions to the same effect were also provided. In addition, students were asked not to go Schematic Knowledge of Probability 18 back to a previous section once they had completed it. They were encouraged to work as quickly and accurately as possible and to ask for assistance if they had any queries. Results Preliminary Analysis Scoring Two types of text-editing scores were derived from Section I: (1) a text-editing score for each category aggregating across problem class and (2) a total text-editing score. For a problem to be scored as correct, students had to classify the problem as belonging to the correct text-editing category and also to identify the missing or irrelevant information where appropriate. The classification score in Section II was the number of correct classifications made over the ten pairs of problems. Partial-credit scoring was used for solution in Section III. A correct answer was scored "2", a partially correct answer was scored "1" (e.g., where students applied a correct formula but made minor calculation errors), and an incorrect answer was scored "0". Each score was converted to a proportion of the maximum possible for that task (text-editing; classification; solution) to facilitate comparisons across tasks. Means, standard deviations, and pairwise correlation for these measures are shown in Table 2. Schematic Knowledge of Probability 19 Table 2 Descriptive statistics and correlations for ( a) text-editing, classification, and solution, and (b) text-editing based on problems class and solution. (a) Variable Mean (sd) (1) (2) (3) (4) (5) TE-missing (1) 0.35 (0.25) TE-irrelevant (2) 0.24 (0.23) .67 TE-sufficient (3) 0.84 (0.22) .30 .24 TE-total (4) 0.48 (0.18) .86 .83 .64 Classification (5) 0.51 (0.18) .41 .26 .18 .37 Solution (6) 0.34 (0.26) .61 .51 .31 .62 .25 (b) Variable Mean (sd) (1) (2) (3) (4) (5) TE-class A-M (1) 0.53 (0.26) TE-class A-NM (2) 0.47 (0.21) .44 TE-class M-I (3) 0.58 (0.36) .46 .37 TE-class M-NI (4) 0.47 (0.16) .21 .10 .16 TE-class Comb. (5) 0.64 (0.33) .54 .32 .55 .27 Solution (6) 0.34 (0.26) .52 .26 .51 .19 .57 p < .05, ** p < .01, *** p < .001; N = 90; A-M = addition-mutually exclusive; A-NM = addition-not mutually exclusive; M-I = multiplication-independent; MNI = multiplication-not independent; Comb = Combination of rules Schematic Knowledge of Probability 20 By way of preliminary analyses, we checked for patterns among the Table 2 means to determine whether performance was in accordance with expectations of text-editing category effects and differences in problem class complexity. As expected, the main-effects for both text-editing category and problem class were significant: Pillai’s Trace = .57, F(2, 86) = 28.25, p < .001; and Pillai’s Trace = .83, F(4, 88) = 219.44, p < .001, respectively. Univariate analyses confirmed the ordering of the text-editing categories with irrelevant problems significantly more difficult than missing problems, F(1,178) = 29.87, p < .001, and both of these more difficult than sufficient problems, F(1,178) = 279.45, p < .001. Regarding the problem classes, problems based on Class A-M (addition rule mutually exclusive events) were significantly easier than problems based on Class A-NM (addition rule events not mutually exclusive), F(1, 356) = 102.47, p < .001; and problems based on Class M-I (multiplication rule independent events) were significantly easier than problems based on Class M-NI (multiplication rule non-independent events), F(1, 356) = 20.64, p < .001. This is consistent with expectations because both A-NM and M-NI entail more complex instantiations of the basic underlying rule. Interestingly, there were no significant differences between A-M and M-I (simple addition and simple multiplication), F(1, 356) = 2.26, p = .13, nor between A-NM and M-NI (complex addition and complex multiplication), F(1, 356) < 1. Continuing these preliminary analyses, we then examined performance on the 10 problems (5 containing sufficient information and 5 containing irrelevant information) that participants were asked to solve in Section III. Results indicated that there was no difference overall in the difficulty of solving problems containing sufficient or irrelevant information (Pillai’s Trace = .03, F(1, 89) = 2.28, p = .135). There was however a significant main effect for problem class (Pillai’s Trace = .20, F(4, 86) = 5.26, p = .001) where problems based on addition rules (A-M and A-NM) were significantly easier to solve than the other problem Schematic Knowledge of Probability 21 classes (M-I, M-NI, and Comb.). As a final step in the preliminary analyses, we then compared the relative difficulty of text-editing and solution as a function of problem class. The solution task is based on only irrelevant and sufficient items and therefore to allow comparison the text-editing score was recalculated without the items that had missing information. The interpretation of the effect of problem class on this new text-editing score remained unchanged. Overall, it was easier to text-edit problems than to solve them (Pillai's Trace = .445, F(1, 89) = 71.25, p < .001). We have already considered the effects of problem class on text-editing and solution separately, so interpretation of these results will not be repeated. Of interest here is the difference between text-editing performance and solution as a function of problem class. The results indicate that although on the whole text editing was easier than solution, this effect was more pronounced for the multiplication and combined problems (M-I, M-NI, Comb: Pillai's Trace = .497, F(1, 89) = 87.91, p < .001) than for the addition problems (A-M, A-NM: Pillai's Trace = .056, F(1, 89) = 5.29, p = .024). In combination with the separate analyses reported above, these findings suggest that text-editing ability is not equally aligned with solution performance across problem classes. Examining the relationship between text-editing and problem solution The main aim of the present study was to test whether the Low and Over (1989, 1990) findings in the field of mathematics regarding the robustness of text-editing as a predictor of solution could be replicated in the field of statistics. Following their method, we used a twostep approach, focusing initially on correlations and regression analysis before moving to a detailed analysis of performance on individual text-editing items containing sufficient and irrelevant information and subsequent performance on the solution of parallel versions (isomorphs) of those same items. The correlations and regression analyses are presented first. Schematic Knowledge of Probability 22 Pairwise correlations among the performance measures are reported in Table 2. The first three variables in Table 2A represent the three text-editing categories. The fourth variable is the total text-editing score, formed by averaging across all items. The fifth variable is the classification task that has been used in past research as a measure of schematic knowledge and was included here as a comparative measure. Table 2B contains text-editing scores based on problem class aggregated across text-editing category. It can be seen from Table 2A that whilst the three text-editing measures had moderate to strong correlations with solution, the relationship between the classification task and solution was much weaker (r = .25, p < .05). To explore the links between this set of predictor variables and solution scores, two multiple regression analyses were conducted. The first analysis explored the relationship between solution and text-editing scores based on sufficient, missing, and irrelevant problems, and the problem-classification task. Together, these variables accounted for 40.5% (37.7% adjusted) of the variability in solution scores (R2 = .405, F(4,85) = 14.489, p < .001). Text-editing problems with missing information was the only measure that contributed uniquely to the prediction of solution score and accounted for nearly one quarter of the total explained variance (sr2 = .097; β = .447, t(85) = 3.72, p <.001). The second regression analysis was for problem class (Table 2B) and therefore focused on a slightly different way to partition the text-editing variance. Together the five variables accounted for 42.2% (38.8% adjusted) of the variation in solutions scores, R = .422, F(5, 84) = 12.28, p < .001. Text-editing of simple addition rule problems (class A-M) uniquely accounted for 4.0% (sr = .04, β = .26, t(83) = 2.40, p =.018), text-editing problems based on simple multiplication rules (class M-I) accounted for 3.5% (sr = .035, β = .23, t(83) = 2.24, p =.027), and text-editing problems based on a combination of simple addition and simple multiplication rules (class Comb) uniquely accounted for 5.6% of the variation in Schematic Knowledge of Probability 23 solution (sr = .056, β = .31, t(83) = 2.84, p = .005). There were no other unique predictors. This tends to indicate that all else being equal, solution is best predicted by variability in textediting problems that entail better-mastered schemas. This is an interesting result particularly when one considers the fact that the simpler rules are typically taught first, followed by instruction in the more complex forms. Hence, schemas for these simple problems are more likely to be better developed than for the more complex forms. The results suggest that the text-editing task was sensitive to variation in these simple types of problems – where knowledge is more likely to be schematic rather than declarative or procedural. To summarise findings regarding text-editing and solution performance at the individual item level, we used the same technique reported by Low and Over (1989, 1990). Table 3 shows the number of students who correctly classified appropriate problems as containing sufficient or irrelevant information (TE+) and who were incorrect in this classification (TE–) relative to their success (+) or failure (–) on solution of the isomorphic problem in Section III. This table considers all events over the ten problems and 90 students. There were many instances where a student was able to correctly text-edit a problem but then failed to solve the structurally equivalent problem. Of the 485 instances where a student was successful in text-editing a problem, there were 334 (68.9%) incorrect attempts at solving a formally identical problem. This is consistent with previous findings of Low and her colleagues showing that text-editing is a necessary but not sufficient condition for solution. Schematic Knowledge of Probability 24 Table 3 Frequency of correct (TE+) and incorrect (TE–) responses for (a) sufficient and (b) irrelevant text-editing problems by solution success (+) and failure (–) on isomorphic problems. (a) Sufficient A-M A-NM M-I M-NI Comb. SOL05 SOL04 SOL10 SOL08 SOL01 + – + – + – + – + – TE+ 26 55 35 43 18 43 21 59 6 73 TE– 0 9 1 11 3 26 0 10 0 11